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;  ABSTRACT  Decades  of  heavy  investment  in  laboratory-based  brain  imaging  and  neuroscience  have  led 
to  foundational  insights  into  how  humans  sense,  perceive,  and  interact  with  the  external  world.  However, 
it  is  argued  that  fundamental  differences  between  laboratory-based  and  naturalistic  human  behavior  may 
exist.  Thus,  it  remains  unclear  how  well  the  current  knowledge  of  human  brain  function  translates  into 
the  highly  dynamic  real  world.  While  some  demonstrated  successes  in  real-world  neurotechnologies  are 
observed,  particularly  in  the  area  of  brain-computer  interaction  technologies,  innovations  and  developments 
to  date  are  limited  to  a  small  science  and  technology  community.  We  posit  that  advancements  in  real- 
world  neuroimaging  tools  for  use  by  a  broad-based  workforce  will  dramatically  enhance  neurotechnology 
applications  that  have  the  potential  to  radically  alter  human-system  interactions  across  all  aspects  of 
everyday  life.  We  discuss  the  efforts  of  a  joint  government- academic-industry  team  to  take  an  integrative, 
interdisciplinary,  and  multi-aspect  approach  to  translate  current  technologies  into  devices  that  are  truly 
fieldable  across  a  range  of  environments.  Results  from  initial  work,  described  here,  show  promise  for 
dramatic  advances  in  the  field  that  will  rapidly  enhance  our  ability  to  assess  brain  activity  in  real-world 
scenarios. 

• 

I  INDEX  TERIVIS  Behavioral  science,  biomarkers,  body  sensor  networks,  brain  computer  interfaces,  brain 
computer  interaction,  data  acquisition,  electroencephalography,  monitoring,  translational  research,  wearable 
sensors. 


I.  INTRODUCTION 

Understanding  human  behavior  and,  at  its  core,  the 
human  brain  has  long  been  considered  critical  to  creating 
revolutionary  technologies.  Over  the  past  three  decades,  sig¬ 
nificant  developments  in  novel  laboratory-based  experimen¬ 
tal  neurotechnologies  have  led  to  meaningful  and  nuanced 
insights  into  the  connection  between  human  experience  and 
nervous  system  structure  and  function;  a  connection  that 
is  considered  to  be  the  foundation  for  understanding  how 
we  sense,  perceive,  and  interact  with  the  external  world 
[1],  [2].  However,  both  theoretical  and  experimental  evi¬ 
dence  suggests  that  there  may  be  fundamental  differences 
in  how  the  human  brain  functions  to  control  behavior  when 
it  is  situated  in  ecologically-valid  environments  (i.e.,  sit¬ 
uated  cognition)  versus  that  observed  in  highly-controlled 
laboratory  environments.  This  generalizability  issue  defines 
the  need  to  expand  the  capability  of  neuroimaging 


technologies  beyond  the  laboratory  and  into  real-world  envi¬ 
ronments  where  natural  human,  task,  and  environmental 
interactions  can  be  studied.  Further,  such  an  extension  of 
neurotechnologies  is  expected  to  have  a  dramatic  impact 
on  the  rise  of  novel  brain-computer  interaction  technologies 
(BCITs;  see  for  discussion:  [3],  [4]).  BCITs  are  potentially 
revolutionary  technologies  that  are  moving  human- systems 
communication  beyond  the  mere  transmission  of  information 
between  man  and  machine  and  toward  the  capability  for 
mutually-derived  data  analysis,  interpretation,  and  prediction. 
The  latter  is  a  core  capability  that  has  the  potential  to  radically 
alter  human-system  interactions  across  all  aspects  of  every¬ 
day  life. 

This  article  discusses  the  current  state  of  neuroimaging 
technologies  and  the  lack  of  any  laboratory-grade  system 
specifically  designed  for  real-world  neuroimaging.  In  large 
part,  real-world  neurotechnology  development  is  limited  by 
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the  fact  that  only  a  relatively  small  circle  of  scientists  and 
technology  developers  are  focused  on  these  issues  and,  related 
to  that,  the  relative  inaccessibility  of  current  neuroimag¬ 
ing  tools  to  the  broad-based  workforce.  Here  we  discuss 
efforts  undertaken  by  the  Army  Research  Laboratory’s  (ARL) 
Translational  Neuroscience  Branch  in  forming  a  govemment- 
academic-industry  team  focused  on  developing  real-world 
neuroimaging  tools  that  support  broad-based  scientific  and 
applied  pursuits.  We  highlight  the  specific  translational  goals 
of  this  effort  and  numerous  advancements  of  this  unique, 
multidisciplinary  team. 

II.  BACKGROUND 

''Brain  imaging  . . .  will  become  even  more  powerful  when 
high-resolution  neural  data  are  gathered  from  participants 
moving  freely  and  interacting  in  natural  ways”  -  Executive 
Office  of  the  President,  National  Science  and 
Technology  Council  [1] 

Neuroimaging  techniques  (e.g.,  electroencephalography 
[EEC],  functional  magnetic  resonance  imaging,  functional 
near-infrared  spectroscopy,  magnetoencephalography,  and 
positron  emission  tomography)  have  unveiled  deep  insights 
into  brain  structures  and  functions  and  how  mental  represen¬ 
tations  and  behavior  are  generated.  However,  many  of  the 
recent  insights  into  the  human  brain  have  been  gained  from 
research  conducted  only  with  highly-controlled  laboratory 
tasks  and  environments  that  are  not  representative  of  the 
real  world.  Although  this  approach  has  advanced  our  basic 
understanding  of  the  brain,  the  extent  to  which  controlled 
laboratory-based  research  generalizes  to  brain  function  under 
the  complex  and  dynamic  environments  of  the  real  world 
is  not  well  understood.  In  fact,  it  has  been  argued  that  the 
consequences  of  failing  to  extend  neuroimaging  approaches 
to  more  realistic  environments  may  be  far  worse  than  fail¬ 
ing  to  generalize  to  natural  environments;  they  may  gener¬ 
ate  fundamental  misunderstandings  of  real-world  brain  func¬ 
tion  [5].  Eor  example,  it  has  been  argued  both  experimentally 
and  theoretically  that  the  complexity  of  the  human  brain’s 
estimated  quadrillion  neural  connections  allows  (or  even 
requires)  not  only  different  individuals  to  process  informa¬ 
tion  differently,  but  also  the  same  individuals  may  engage 
different  brain  regions  to  cognitively  process  similar  infor¬ 
mation,  particularly  in  response  to  contextual  changes  [2], 
[6].  These  concepts  suggest  that  there  may  be  fundamental 
differences  in  situated  cognition  relative  to  that  observed  in 
highly  controlled  laboratory  environments,  supporting  the 
need  for  more  ecological  approaches  [7]  focusing  on  human, 
task,  and  environmental  interactions  [8].  Our  primary  motiva¬ 
tion  for  attempting  neuroimaging  in  real-world  environments 
is  based  on  the  potential  impact  that  answering  these  open 
fundamental  scientific  questions  would  have  on  translating 
basic  neuroscience  to  technology  development. 

Initial  attempts  at  real-world  neuroimaging  were  focused 
on  controlling  and  replicating  aspects  of  naturally  occur¬ 
ring  human  behaviors  within  the  laboratory  [9]-[14]. 


More  recently,  attempts  have  been  made  to  extend  neuroimag¬ 
ing  to  more  complex  environments  previously  thought  to 
be  unapproachable  even  with  state-of-the-art  methodologies 
[15]-[19].  These  efforts  have  generally  adapted  laboratory- 
based  neuroimaging  technologies  to  suit  specific  and  unique 
measurement  requirements.  While  they  have  made  signif¬ 
icant  steps  forward,  the  overall  field  is  still  quite  limited; 
researchers  are  forced  to  balance  between  studying  well- 
controlled,  potentially  unnatural,  behavior  and  performing 
studies  that  generalize  to  ecologically  relevant  situations.  In 
large  part,  the  research  is  limited  by  the  technological  gap 
between  the  neuroimaging  tools  that  currently  exist  and  the 
tools  needed  to  appropriately  conduct  neuroimaging  research 
in  real-world  environments.  The  development  of  such  tools 
will  require  more  than  a  mere  extension  of  current  laboratory- 
based  technologies  due  to  the  complex  dynamics  and  uncer¬ 
tainties  inherent  to  real-world  environments  combined  with 
an  overwhelming  number  of  influences  that  limit  signal- 
to-noise  ratio  [15],  [19]-[21].  Recently,  significant  progress 
has  been  made  in  novel  technologies  that  have  the  potential 
to  radically  advance  real-world  neuroimaging  (for  reviews, 
see  [4],  [22]).  While  these  advancements  are  enabling 
preliminary  system  designs,  true  real-world  neuroimaging 
will  require  further  development  in  four  critical  technology 
areas:  Neuroimaging  Hardware  including  multi-aspect  sen¬ 
sors  arrays,  power,  and  on-board  processing  capabilities  that 
meet  the  wearability  and  usability  requirements  for  real-world 
research  and  application;  Algorithms  addressing  issues  rang¬ 
ing  from  signal-to-noise  ratios  to  multi-aspect  sensor  integra¬ 
tion  enabling  the  joint  interpretation  of  neural  signals,  human 
behavior,  and  environmental  context;  Interfaces  that  both 
facilitate  highly  multidisciplinary  research  amongst  scientists 
from  varying  fields  of  study  and  enable  effective  data  collec¬ 
tion  by  naive  participants;  and  Experimentation,  Testing,  and 
Validation  including  novel  paradigms  for  neuroimaging  under 
uncertainty  and  technologies  for  validating  new  hardware  and 
algorithms. 

III.  TRANSLATIONAL  RESEARCH  TOOLS 

"Agencies  should  continue  to  develop  new  tools  and 
techniques  for  research  and  ensure  the  broadest 
dissemination  possible  for  those  tools.”  -  Executive  Office  of 
the  President,  National  Science  and  Technology  Council  [1] 

ARL’s  Translational  Neuroscience  Branch  has  formed  a 
govemment-academic-industry  team  to  develop  a  wide  vari¬ 
ety  of  novel  tools  to  enable  real-world  neuroimaging  systems 
across  a  wide  variety  of  operationally-relevant  environments 
and  in  support  of  neurotechnologies,  such  as  BCITs.  Such 
tools  will  directly  support  enhancing  human- system  inte¬ 
gration  with  the  potential  for  broad-based  impacts  across 
almost  all  aspects  of  everyday  life.  However,  traditional  neu¬ 
roscience  approaches  to  tool  development  have  the  potential 
for  creating  the  same  translational  risks  that  have  hindered  the 
transition  of  neuroscience  research  in  the  past.  Eor  example, 
the  broad  variability  in  technical  system  requirements  for 
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different  applications  (e.g.,  temporal  and  spatial  resolution; 
amounts,  types,  and  sources  of  data;  acceptable  signal-to- 
noise  ratios)  combined  with  the  sheer  complexity  of  the  ner¬ 
vous  system  and  the  uncertainty  of  the  signals  being  detected 
can  lead  to  an  abundance  of  research  results  and  developed 
tools  that,  while  important  in  and  of  themselves,  ultimately 
cannot  be  effectively  combined  into  singular  functional  sys¬ 
tems.  Furthermore,  the  development  and  application  of  neu¬ 
rotechnologies  remains  limited  to  a  relatively  small  circle  of 
scientists  and  technology  developers  and  has  not  achieved 
a  broader  focus  within  the  greater  science  and  technology 
(S&T)  development  community.  Complicating  this,  the  large 
majority  of  existing  tools  are  expensive  and/or  require  scien¬ 
tists  with  an  extensive  knowledge  base  to  use  them  appropri¬ 
ately.  As  long  as  the  tools  and  knowledge  needed  to  develop 
neurotechnologies  remains  inaccessible  to  the  broad-based 
workforce  within  the  S&T  community,  innovation  and  devel¬ 
opment  will  remain  limited,  especially  with  regards  to  BCITs. 

To  mitigate  these  issues,  it  is  critical  to  take  a  multi¬ 
disciplinary  approach  in  which  scientists  and  engineers  from 
a  wide-range  of  backgrounds  work  together  to  push  neu¬ 
roimaging  systems  into  the  real  world  in  both  scientific 
and  applied  pursuits.  We  focus  on  coupling  the  creation  of 
functional  systems  targeted  at  narrow  specific  aims  along 
with  the  development  of  tools  targeted  at  enabling  a  multi¬ 
disciplinary  team  to  bridge  the  scientific  and  technological 
gaps  between  neuroscientists,  signal  processing  experts,  and 
technology  innovators,  among  others  [23].  We  posit  that  the 
results  from  these  kinds  of  functional  systems,  even  when 
narrowly  focused  on  specific  questions  and  applications,  have 
the  potential  to  dramatically  change  how  we  conceive  of 
human  brain  functioning  and  its  interactions  with  real-world 
technologies,  providing  insights  that  will  directly  influence 
the  types  and  functionality  of  the  real-world  neuroimaging 
tools  being  developed. 

With  this  in  mind,  ART  has  developed  two  translational 
research  objectives,  described  below,  that  target  develop¬ 
ing  capabilities  to  answer  fundamental  scientific  questions 
in  real-world  environments  and  applying  that  knowledge  to 
proof-of-principle  neuro technologies.  Both  objectives  will 
require  wireless,  high-resolution,  ruggedized,  wearable  hard¬ 
ware  and  software  tools  that  disentangle  and  differentiate 
brain  signal,  human  behavior,  environmental  events,  and  arti¬ 
facts.  However,  the  relative  requirements  for  each  objec¬ 
tive  are  very  different.  Pursuing  these  objectives  is  a  team 
comprised  of  members  from  different  collaborative  programs 
(e.g..  Cognition  and  Neuroergonomics  Collaborative  Tech¬ 
nology  Alliance  (CaN  CTA;  [24]),  the  Institute  for  Collab¬ 
orative  Biotechnologies  (ICB),  Small  Business  Innovation 
Research  Awards  (SBIR),  and  internal  research  programs, 
among  others). 

A  OBJECTIVE  J:  LABORATORY  GRADE  REAL-WORLD 
NEUROIMACINC 

Our  first  objective  is  the  creation  of  a  toolkit  that  can  support 
basic  research  on  fundamental  questions  underlying  neural 
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FIGURE  1.  Prototype  Real-World  Neuroimaging  System.  Four  components 
are  illustrated,  an  Android  device  for  data  collection  and  integration,  a 
laboratory  grade,  dry  electrode,  wirelesss,  high  resolution  EEC  system,  a 
Bioharness  3,  and  an  Affectiva  Qsensor.  Android  screenshots  depict  some 
examples  of  activity  logging  components,  such  as  a  start/stop  logger 
(left),  two  self-assessment  surveys  (middle),  and  a  visuomotor  continuous 
tracking  task  (right). 


functions  within  real-world  environments  (e.g.,  see  Figure  1). 
In  the  near-term,  the  development  of  this  hardware  and  soft¬ 
ware  toolkit  targets  specific  questions  regarding  how  the 
interrelated  factors  of  stress,  fatigue,  and  circadian  rhythms 
impact  persons  within  real-world  office  environments.  As 
compared  to  state-of-the-art  laboratory  neuroimaging  tools, 
the  characteristics  of  the  target  system  include:  similar  tem¬ 
poral  and  spatial  resolution  neural  data;  the  capacity  for 
extensive  multi-aspect  sensor  integration,  including  physio¬ 
logical,  behavioral,  and  environmental  sensing  to  help  build 
a  rich  characterization  of  context;  extended  duration  of  data 
acquisition  system  wearability;  greater  ruggedization;  limited 
set-up  time  requirements;  usability  by  both  subject  matter 
experts  and  trained  participants;  functionality  in  office  type 
environments;  efficient  data  storage,  transfer,  and  handling; 
and  complex,  often  computationally-intensive  offline  data 
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analysis  by  scientists  from  a  variety  of  neuroscience  and 
engineering  backgrounds. 

B.  OBJECTIVE  2:  REAL-TIME  NEUROIMACINC  FOR  BCIT 

The  second  objective  is  to  develop  a  hardware  and  soft¬ 
ware  toolkit  that  can  support  the  development,  assessment, 
and  application  of  BCITs  in  real-world  settings.  Near-term 
objectives  focus  on  designing  a  system  specifically  in  support 
of  the  development  and  assessment  of  fatigue-based  perfor¬ 
mance  decrements  and  target-identification  brain-computer 
interaction  technologies  in  real-time.  Compared  to  state-of- 
the-art  laboratory  neuroimaging  tools,  in  the  near-term  the 
development  of  this  hardware  and  software  toolkit  will  target: 
more  narrowly  defined  neural  signals  to  support  application 
functionality;  the  capability  for  multi-aspect  sensor  integra¬ 
tion  to  help  build  context;  greatly  extended  duration  of  data 
acquisition  system  wearability  to  meet  expected  application 
use;  full  ruggedization;  limited  to  no  set-up  time  require¬ 
ments;  usability  by  technicians  with  limited  experience  and 
naive  participants;  functionality  across  the  wide  variety  of 
environments  in  which  the  applications  are  expected  to  be 
deployed;  and  capability  for  real-time  analysis  with  efficient 
storage,  transfer,  and  handling  of  raw  data  to  enable  further 
application  R&D. 

IV.  REAL-WORLD  NEUROIMACINC  SYSTEMS 

The  conceptual  and  theoretical  foundations  for  the  real- 
world  neuroimaging  approach  we  are  advocating  here  can  be 
found  in  scientific  disciplines  ranging  from  artificial  intel¬ 
ligence  and  robotics  [25]  to  embedded  and  embodied  or 
situated  cognition  [26]  to  ecological  psychology  [27]-[29]. 
The  growing  importance  of  these  concepts  is  reflected  in  an 
increasing  appreciation  that  human  mental  processes  are  inti¬ 
mately  bound  with  our  interactions  with  the  world  [30],  [31]. 
Considering  this  perspective,  ART  and  its  partners  advo¬ 
cate  an  approach  to  real-world  neuroimaging  that  lever¬ 
ages  technological  advancements  across  a  broad  array  of 
sensors  that  sample  not  only  the  human  brain,  but  human 
physiology  and  behavior,  in  addition  to  task  and  environ¬ 
mental  conditions  [20],  [32],  [33].  A  fundamental  element 
of  this  multi-aspect  approach  to  neuroimaging  is  to  place 
information  about  brain  function  into  the  context  of  when 
it  occurs.  In  typical  laboratory  settings,  this  is  accomplished 
through  control  of  the  participant’s  actions  and  environment; 
however,  such  control  is  infeasible  in  real-world  environ¬ 
ments  and  situations. 

Over  the  past  five  years,  there  have  been  tremendous 
advancements  in  sensor  and  communication  technologies  that 
are  enabling  our  vision  of  real-world  neuroimaging.  Perhaps 
the  most  significant,  direct  advancements  have  occurred  in 
the  area  of  dry  and  wireless  EEG  systems.  Of  the  array 
of  existing  classes  of  neuroimaging  technologies,  EEG  has 
shown  the  most  promise  as  a  near-term  solution  in  terms  of 
quality,  mobility,  and  wearability  within  real-world  environ¬ 
ments  and,  as  such,  has  seen  the  most  commercial  devel¬ 
opment  in  this  domain.  A  second  area  of  advancement  has 


been  in  EEG  analysis  software  that  is  much  more  broadly 
available,  ranging  from  commercial  ventures  (e.g.,  BESA, 
MidWare  Technologies,  Neuroscan)  to  freely  available  tool¬ 
boxes  available  from  academic  research  groups  (e.g.  Eield- 
Trip,  Brainstorm,  EEGLab,  ERPLab),  to  academic  software 
which  contains  real-time  analysis  and  classification  capabil¬ 
ities  (e.g.,  BCILAB  toolbox  developed  by  the  University  of 
California,  San  Diego  (UCSD);  [34],  [35]  or  BCI2000  from 
the  Wadsworth  Center  of  the  New  York  State  Department  of 
Health;  [36]),  as  well  as  sophisticated  approaches  to  extract¬ 
ing  neural  signals  in  complex  and  noisy  environments  (e.g., 
see  DETECT  and  EEGVIS  toolboxes  developed  by  the  Uni¬ 
versity  of  Texas  at  San  Antonio  (UTS  A)  in  collaboration  with 
ARE;  [37]-[39]).  These  advancements  in  neuroimaging  tech¬ 
nologies  provide  a  strong  basis  for  leveraging  the  broad  rang¬ 
ing  sensor  advancements  that  have  occurred  in  other  domains 
to  create  multi-aspect  systems  that  meet  our  team’s  objectives. 

A.  LABORATORY  GRADE  REAL-WORLD  NEUROIMACINC 

Laboratory-grade  systems  designed  specifically  for  real- 
world  neuroimaging  do  not  exist  to  our  knowledge. 
Current  laboratory-grade  hardware  solutions  are  bulky  and 
not  designed  for  comfort  or  ease-of-use.  Eor  example,  wired 
systems  with  as  many  as  256  sensor  leads  are  awkward  to 
wear,  take  long  set-up  times  of  up  to  an  hour,  and  can  cause 
participant  discomfort  when  worn  over  extended  durations 
(Eigure  2).  The  typical  wet  electrodes  found  in  laboratory 
equipment  can  begin  to  dry  out  within  30-minutes  to  2  hours, 
which  directly  influences  signal  quality  [40].  High-bandwidth 
data  transmission  requirements  typically  force  participants  to 
be  tethered  to  computing  systems  or  to  carry  relatively  heavy 
hardware,  such  as  batteries,  amplifiers,  and  laptop  computers 
[16].  These  hardware  constraints  limit  the  naturalistic  behav¬ 
iors  that  can  be  observed,  as  well  as  the  types  of  contexts  that 
may  be  investigated. 

Despite  these  limitations,  there  have  been  several  recent 
efforts  to  understanding  brain  function  in  real-world  condi¬ 
tions  using  actual  laboratory  systems.  One  approach  has  been 
to  take  laboratory-grade  EEG  systems  into  relatively  well- 
controlled,  real-world  task  environments,  such  as  on-road 
vehicle  driving  [41],  learning  marksmanship  [21],  [42]  or 
even  flying  a  plane  [43].  These  types  of  research  efforts  allow 
the  study  of  the  rich  task  dynamics  and  behavioral  contexts  of 
real-world  conditions,  while  taking  advantage  of  behaviors 
that  do  not  generally  require  a  lot  of  bodily  movements  that 
would  induce  excessive  motion  and/or  muscle-related  signal 
artifacts  that  negatively  affect  the  quality  of  the  recorded 
raw  EEG  signals.  However,  the  constraints  imposed  by  the 
hardware  necessarily  limit  this  approach  to  a  very  narrow 
scope  of  tasks  and  experimental  durations.  Eurther,  even  in 
these  real-world  environments,  participants  are  so  constrained 
by  the  systems  that  it  is  unclear  how  effective  the  results 
will  be  in  generalizing  to  other  types  of  naturalistic  human 
behavior. 

An  alternative  approach  has  been  to  take  laboratory-grade 
measurement  and  tasks  into  environments  that  reflect  rele- 
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FIGURE  2.  State-of-the  Art  Laboratory  Grade  EEC  System  made  "Mobile." 
Participant  wears  a  high-density  wet-electrode  cap  which  is  wired  to  a 
laptop  and  amplifiers  in  a  backpack.  This  system  takes  approximately  1 
hour  to  set-up,  and  can  become  uncomfortable  in  minutes. 

vant  aspects  of  real-world  settings.  For  example,  ARL  stud¬ 
ied  standard  auditory  event-related  potentials  (ERP)  while 
participants  stood,  walked,  or  ran  on  a  treadmill  or  sat 
on  a  six-degree-of-freedom  ride  motion  simulator  (RMS) 
during  increasingly  dynamic  ride-motion  conditions  [19]. 
The  motion  conditions  of  this  study  extended  the  behav¬ 
ioral  environments  where  ERP  responses  were  previously 
observed  far  beyond  typical  laboratory  conditions,  while 
still  structuring  the  environment  in  ways  consistent  with 
controlled-laboratory  research:  the  treadmill  allowed  the  con¬ 
trol  of  walking  or  running  speed;  the  RMS  ensured  that 
all  participants  experienced  the  exact  same  ride  motions. 
Except  under  the  most  extreme  motion  conditions,  the  results 
demonstrated  that  even  when  large  body  movements  induced 
significant  motion-  and  muscle-related  signal  artifacts,  well- 
known  temporal  (e.g.,  latencies  of  NlOO  and  P300  peaks)  and 
spatial  (e.g.,  increased  P300  amplitudes  from  anterior  to  pos¬ 
terior  electrode  sites)  patterns  of  ERP  responses  seen  in  the 
laboratory  can  be  recovered.  In  a  similar  fashion,  team  part¬ 
ners  at  the  University  of  Michigan  (UM)  and  UCSD  also 
demonstrated  that  visual  ERPs  could  similarly  be  recov¬ 
ered  during  treadmill  walking  using  Independent  Component 
Analysis  (ICA)  techniques  [15].  UM  further  collaborated 
with  ARL  to  show  that  visual  event-related  activity  could 
be  observed  using  artifact-resistant  functional  connectivity 
metrics  [44].  Though  these  experiments  lack  the  richness 


and  depth  of  real-world  neuroimaging,  they  demonstrate  the 
plausibility  of  extracting  traditional  neural  signatures  in  the 
face  of  significant  artifact. 

Starting  from  these  initial  studies,  our  group  has  contin¬ 
ued  to  build  towards  more  fully  realized  real-world  neu¬ 
roimaging  capabilities.  First,  by  extending  our  efforts  into 
more  complex  task  environments  through  the  use  of  complex 
and  behaviorally- sophisticated  virtual  reality  simulations 
(cf.,  [45]-[47]),  we  have  shown  that  it  is  possible  to  suc¬ 
cessfully  extend  current  neuroscience  research  paradigms  to 
better  capture  and  reflect  the  subtle  complexities  of  real- 
world,  embedded,  interaction.  Second,  by  developing  more 
mobile  and  user-friendly  measurement  systems  we  hope  to 
both  minimize  the  burden  on  the  experimenter  and  to  alleviate 
intrusiveness  on  the  participant,  thereby  encouraging  more 
naturalistic  behavior. 

1)  OVERVIEW:  STRESS  AND  FATIGUE  NEUROIMAGING 
SYSTEM  VI  .0 

Currently,  ARL  and  its  partners.  National  Chiao  Tung 
University  (NCTU)  in  Taiwan  and  DCS  Corporation  (DCS) 
are  collaborating  to  implement  a  first-of-its-kind,  wearable, 
high-density,  real-world  neuroimaging  system  for  studying 
stress  and  fatigue  in  the  workplace  [48].  This  laboratory- 
grade  Stress  and  Fatigue  Neuroimaging  (SFN)  System  Vl.O 
has  been  specifically  designed  to  overcome  several  of  the 
limiting  issues  for  real-world  neuroimaging.  The  prototype 
is  comprised  of  an  Android  device,  a  Samsung  Galaxy 
S3  in  the  current  implementation,  which  monitors,  records, 
and  synchronizes  data  streamed  from  three  physiologi¬ 
cal  monitoring  devices:  a  high-density  MINDO  EEG  sys¬ 
tem  (Hsinchu,  Taiwan),  a  multifunction  Zephyr  Bioharness 
3  lightweight  chest  strap  (Annapolis,  MD,  USA),  and  a 
multifunction  Affectiva  QSensor  wrist-watch  style  device 
(Waltham,  MA,  USA);  as  well  as  obtaining  user  inputs 
(Figure  1).  All  three  of  these  physiological  systems  use  dry- 
type  electrodes  for  faster  set-up  and  longer-term  wearabil¬ 
ity.  The  complete  system  weighs  approximately  406  grams 
(0.9  pounds;  MINDO-64:  200g,  Bioharness  3:  50g,  QSensor: 
22. 7g,  Samsung  Galaxy  S3:  133g).  The  numerous  compo¬ 
nents  of  the  prototype  system  have  been  integrated  and  the 
system  is  undergoing  initial  phases  of  use-case  testing. 

As  this  system  is  designed  for  scientific  pursuits,  the 
primary  analysis  software  will  be  offline,  where  the  Android 
based  physiological  data  and  behavioral  data  can  be  com¬ 
bined  with  contextual  information  from  additional  sources, 
such  as  the  user’s  calendar,  user  annotations,  or  questionnaire 
responses  (see  below  for  more  detailed  description).  State-of- 
the-art  offline-analyses  programs,  such  as  EEGLAB,  will  be 
used  for  data  processing. 

2)  LONG-TERM,  WEARABLE  NEURAL  DATA  ACQUISITION 
SYSTEM 

The  centerpiece  of  SEN  System  V 1 .0  is  the  NCTU-developed 
64-channel  wireless  EEG  system  (MINDO-64),  which  is 
designed  to  address  issues  of:  high-resolution  laboratory- 
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grade  data  acquisition,  long-term  comfortable  wear,  quick 
user  set-up,  and  high  portability.  The  use  of  high-bandwidth 
wireless  data  transmission  overcomes  the  constraints  of  phys¬ 
ically  tethering  participants  to  computing  systems  or  hav¬ 
ing  them  carry  the  equipment  with  them  during  the  experi¬ 
ment.  The  MINDO-64  system  uses  both  Bluetooth  and  WiFi 
modules  to  transmit  EEG  signal  during  recording,  offering  a 
maximum  512Hz  sampling  rate  with  24-bit  resolution  and  is 
the  first  wireless  EEG  system  integrated  into  a  form  factor 
with  a  flexible  printed  circuit  board  inside,  a  novel  head 
circumference  adaptable  mechanical  design  for  improved  sta¬ 
bility,  and  active  dry  sensors  that  amplify  signals  at  a  very 
early  stage  to  improve  signal-to-noise  ratios  and  avoid  the 
need  for  skin  preparation  and  gel  application.  The  system  also 
stores  raw  data  on-board  for  use  in  off-line  processing  to  help 
mitigate  critical  issues  surrounding  wireless  transmission. 
Through  the  integration  of  active  sensor  and  power  control  on 
the  main  circuit,  the  system  enables  long-term  wear  of  up  to 
10  consecutive  hours  of  operation  time.  The  system’s  wireless 
technologies,  light  weight  (200g),  and  dry  sensor  designs  also 
support  comfort,  fast  set-up,  and  portability. 

3)  UNDERSTANDING  CONTEXT 

One  of  the  most  critical  assumptions  in  our  vision  of 
real-world  neuroimaging  is  that  developing  a  rich  multi¬ 
dimensional  characterization  of  context  will  be  critical  to  the 
accurate  and  meaningful  interpretation  of  measured  neural 
activity.  As  outlined  above,  most  previous  attempts  at  neu¬ 
roimaging,  both  traditional  and  mobile,  have  relied  on  con¬ 
trolling  context  through  environmental,  behavioral,  and  task 
constraints,  which  can  sacrifice  naturalistic  human  behav¬ 
ior.  By  contrast,  we  posit  that  meaningful  interpretation 
of  observed  real-world  neural  signals  will  result  from  the 
characterization  and  understanding  of  statistical  relationships 
between  broadly  defined  contextual  events  (to  include  behav¬ 
ioral,  physiological,  task,  and  environmental  events  across 
multiple  timescales)  and  the  observed  neural  activity  that 
supports  our  behavior. 

Working  in  collaboration,  ART  and  DCS  developed  a 
range  of  applications  on  the  Android  platform  to  enable  an 
observational,  multi-aspect  measurement  approach  targeted 
at  building  a  context  to  interpret  the  neural  activity  related  to 
stress  and  fatigue.  In  building  the  context  of  the  internal  state 
of  the  user,  the  system  integrates  the  Bioharness  for  EGG 
and  respiration  rate;  the  QSensor  to  provide  skin  conduc¬ 
tance  (a  form  of  electrodermal  activity)  and  skin  temperature 
data;  and  subjective  reports  related  to  stress  and  fatigue.  To 
build  a  context  of  the  user’s  activities,  the  system  integrates 
3-axis  accelerometers  from  both  the  Bioharness  and 
QSensor  with  incident  reports  (e.g.,  voluntary  annotation  of 
a  stressful  event),  start  and  end  times  of  defined  user  activi¬ 
ties  (eating,  drinking,  meetings,  exercise,  etc.)  questionnaire 
responses,  and  other  user  annotations. 

Off-line  software  will  be  used  to  further  integrate  the 
Android-based  data  with  information  from  the  user’s  calendar 
and  other  information  gathered  by  the  experimenters.  EEG 


analysis  software  has  also  been  used  to  isolate  and  inter¬ 
pret  EEG  activity  that  is  traditionally  discarded  as  artifact 
(e.g.,  muscle  activity;  see  description  of  artifact  classifiers 
by  Lawhern  and  colleagues  in  Section  IV).  Our  multi-aspect 
approach  advocates  integrating  these  different  sources  of 
information,  such  as  artifact,  to  complete  a  contextual  picture 
that  is  sufficient  to  frame  the  moment-to-moment  dynamics 
of  the  user’s  brain  activity. 

4)  OTHER  IMPORTANT  ISSUES 

Several  other  features  have  been  designed  into  SEN  System 
Vl.O  to  address  issues  critical  to  real-world  neuroimaging. 
Eor  example,  the  wearer  of  a  real-world  EEG  system  is  going 
to  have  more  responsibility  for  the  system  and  for  the  data 
collection  than  in  typical  experiments  where  the  experimenter 
is  present.  Towards  this  end,  novel  user  interfaces  have  and 
continue  to  be  designed  to  enable  trained  users  to  set-up  the 
system  and  troubleshoot  bad  sensors  and  periods  of  ineffec¬ 
tive  data  collection.  A  second  critical  issue  is  the  need  to 
reference  observational  findings  to  the  extant  literature  on 
stress  and  fatigue.  To  address  this  issue,  our  team  developed 
Android-based  applications  that  represent  laboratory  tasks 
including  a  continuous  tracking  task  [49]  and  can  be  triggered 
through  a  master  scheduler.  A  third  issue  is  the  observer 
effect;  the  concept  that  the  act  of  measurement  inherently 
changes  the  phenomena  being  measured.  While  no  system 
can  avoid  the  observer  effect,  the  new  technology  solutions 
described  here  help  to  minimize  some  of  the  most  obvious 
and  intrusive  effects  that  have  undoubtedly  affected  the  mea¬ 
surement  of  human  behavior  in  previous  research.  Einally, 
SEN  System  Vl.O  has  been  designed  to  be  highly  flexible, 
enabling  the  researcher  to  choose  nearly  any  combination  of 
components  and  applications,  and  thus  weigh  the  costs  and 
benefits  of  the  system  to  measure  true  naturalistic  behavior  in 
addressing  their  research  questions. 

B.  REAL-TIME  NEUROIMACINC  FOR  BCIT 

Many  of  the  critical  issues  for  real-world  neuroimaging  are 
discussed  above  and  are  being  addressed  in  the  design  of 
SEN  System  Vl.O.  However,  there  are  critical  differences  in 
designing  such  a  system  for  applied  as  opposed  to  scientific 
pursuits.  Recently,  there  have  been  tremendous  advances  in 
commercial  EEG  technologies,  and  many  systems  start  to 
address  the  issues  specific  to  applied  systems.  In  this  section 
we  discuss  some  of  those  advances  in  light  of  four  critical 
issues  for  applied  systems: 

1)  BUILT  FOR  A  NARROW  PURPOSE 

Scientific  systems  are  generally  intended  for  broad-ranging 
research,  where  many  of  the  specifics  of  the  signals  elicited 
through  their  use  are  unknown  to  the  system  designers 
a  priori.  As  a  result,  systems  are  generally  designed  to  maxi¬ 
mize  data  availability  for  the  user.  Neuroimaging  for  BCIT 
purposes,  however,  is  almost  exclusively  framed  around  a 
previously  established  target  signal,  or  at  least  a  specific  realm 
of  signal  types.  As  a  result,  BCIT  systems  can  be  designed 
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efficiently  to  detect  the  target  signal,  at  the  cost  of  flexibility 
or  maximal  data  availability.  For  example,  systems  could  be 
designed  to  use  small  numbers  of  electrodes  placed  in  strate¬ 
gic  locations,  as  opposed  to  the  high-density  arrays  found 
in  laboratory-grade  systems.  Likewise,  designers  of  specific 
BCIT  systems  will  have  to  weigh  the  costs  and  benefits  of 
addressing  many  constraints  such  as:  a)  the  additional  com¬ 
plexity  required  by  high-precision  timing  accuracy  or  timing 
integration  with  several  other  devices,  b)  the  ability  to  flexibly 
change  the  devices  to  which  they  are  connected,  c)  the  need 
for  additional  sensory  modalities  to  supplement  EEG,  and 
d)  the  cost  and  effectiveness  of  manufacturing  processes. 

^  n 
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/ 
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FIGURE  3.  An  early  prototype  system  for  BCITs.  The  NCTU-developed 
IVIINDO-4  system  for  potential  BCITs  that  detect  and  mitigate 
fatigue-related  performance  decrements  during  driving.  System  uses  soft 
pads  (U.S.  penny  and  pen  shown  for  scale)  that  snap  into  a  headband, 
and  streams  data  via  Bluetooth  for  easy  system  integration. 

How  to  optimally  create  tools  that  both  enable  the  devel¬ 
opment  of  such  narrowly  focused  systems  and  assist  in 
such  cost/benefit  analyses  remains  an  open  question.  Our 
team’s  current  research  efforts  focus  on  the  development  of 
cost-effective,  dedicated,  integrated  components  for  sensing 
specific  signals  and  developing  prototypes  suitable  for  assess¬ 
ing  fatigue-based  performance  decrements,  while  also  main¬ 
taining  enough  flexibility  to  assure  easy,  rapid  integration 
between  the  neuroimaging  component  and  potential  BCIT 
systems.  Eor  example,  one  of  ARL’s  first  partnerships  in 
neurotechnologies  with  NCTU  resulted  in  the  MINDO-4 
(see  EigureS;  [50],  [51]). 

2)  REAL-TIME  PROCESSING 

Perhaps  the  most  critical  constraint  on  BCITs  is  the  require¬ 
ment  for  real-time  or  near-real  time  processing.  Despite  the 
many  approaches  and  advances  in  EEG  analysis,  real-time 
signal  processing  for  online  applications  is  one  area  that  still 
receives  relatively  little  attention.  Currently,  the  most  signifi¬ 
cant  work  in  this  area  is  coming  from  the  groups  developing 
actual  BCITs  [52].  Our  team’s  efforts,  described  primarily  in 
section  V,  attempt  to  integrate  and  extend  these  efforts  into  a 
systems-level  view. 


3)  USABLE  BY  ANYONE 

Euture  BCITs  are  envisioned  as  a  core  technology  that  can 
permeate  all  aspects  of  everyday  life  [3].  As  such,  it  will  be 
critical  that  the  systems  are  usable  by  anyone.  Consequently, 
the  systems  must  be  simple  to  setup  and  troubleshoot  and  they 
likely  will  have  to  be  relatively  transparent  or  even  invisible 
to  the  operator  for  compliance  purposes.  Eor  example,  games 
such  as  the  Star  Wars  Eorce  Trainer  (Uncle  Milton)  and  the 
Mindflex  (Mattel)  marked  incredible  advances  in  making 
BCITs  usable;  however,  they  are  still  far  from  being  a  reliable 
neuroimaging  technology  that  people  will  wear  regularly  and 
for  long  durations  in  everyday  environments. 

One  of  the  critical  elements  is  the  lack  of  systems 
and  tools  to  enable  non-specialists  to  develop  and  assess 
BCITs  [23].  An  important  step  towards  this  goal  was 
the  release  of  the  commercial  product,  the  Emotiv  EPOC 
(San  Erancisco,  CA,  USA).  This  system  is  relatively  easy 
to  set-up  and  can  allow  non-specialists  to  use  classifiers  to 
train  the  system  to  control  keyboard  keys  among  other  things. 
However,  the  ease  of  use  of  the  system  without  sufficient 
software  for  basic  developer  education  and  training  can  lead 
to  misinterpretation  of  system  outputs  and  is,  potentially,  a 
significant  limiter  on  BCIT  development  [23];  current  sys¬ 
tems  allow  developers  to  make  basic  mistakes  that  will  ulti¬ 
mately  inhibit  the  BCIT’s  applicability.  Team  partners  at 
USCD  have  also  released  an  open-source  BCIT  toolkit  for 
MATLAB  that  aids  in  bridging  the  gaps  between  specialists 
from  different  backgrounds  (e.g.,  neuroscientists,  engineers, 
psychologists),  BCILAB  [53].  Eurther  significant  advance¬ 
ments  are  necessary  to  fully  bridge  the  gap  to  non-specialists. 

4)  SAFE,  ROBUST,  AND  COST  EFFECTIVE 

The  real-world  neuroimaging  systems  developed  for  BCITs 
will  need  to  be  cost-effective,  as  well  as  safe  and  robust 
across  a  wide  variety  of  environments,  including  some  that 
are  highly  dynamic  and  potentially  hazardous.  While  these 
are  issues  for  the  scientific  systems  described  above,  as  well, 
there  are  several  factors  that  emphasize  their  importance  for 
applied  systems:  Safety  is  a  critical  issue  for  any  system  with 
sensors,  electronics,  and  other  components  directly  above  the 
scalp.  Health  and  cleanliness  issues  must  be  addressed  by 
all  system  designs.  In  the  scientific  domain,  constraints  are 
placed  on  experimentation  to  limit  the  risk  to  study  partici¬ 
pants,  such  as  those  emplaced  by  Institutional  Review  Boards. 
However,  as  applied  systems  become  more  functional  and 
available,  they  are  likely  to  be  found  in  situations  with  the 
potential  for  trauma-inducing  head  impact.  As  discussed  in 
section  V,  ARE  and  NCTU  are  investigating  solutions  to 
enable  safe  system  designs  for  all  environments. 

Systems  will  also  have  to  be  robust  and  cost-effective. 
Robust  systems  will  need  to  operate  consistently  and  reli¬ 
ably  within  an  environment  that  could  be  physically  dam¬ 
aging  to  a  sensitive  unit,  as  well  as  being  able  to  properly 
deal  with  the  noise  and  artifacts  that  will  occur  within  the 
BCIT  application.  Eor  example,  a  system  designed  for  use 
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within  an  off-road  vehicle  must  be  capable  of  functioning  for 
extremely  long  time  periods  when  subject  to  vibration  (e.g., 
see  [19]),  as  well  as  electrical  noise  from  the  engine  and  other 
sources.  Further,  the  systems  must  be  available  at  a  cost  that  is 
affordable  to  a  wide  range  of  potential  users  and  BCIT  devel¬ 
opers.  Alternative  approaches  to  cost-effective  robust  EEG 
measurement  range  from  systems  with  very  low-cost  dispos¬ 
able  sensors  to  higher-quality,  highly -reconfigurable  systems 
with  lower  sensor  density.  ARE  is  currently  working  with 
the  Defense  Advanced  Research  Projects  Agency  (DARPA) 
on  its  efforts  to  advance  portable,  inexpensive,  easy-to-use 
hardware  solutions  to  EEG  (e.g.,  see  DARPA  proposed  SBIR 
SB  1 3 1  -002).  In  section  V  below,  we  highlight  several  specific 
algorithmic  approaches  enabling  robust  system  design. 

V.  ENABLING  TECHNOLOGIES  FOR  FUTURE  SYSTEMS 

As  discussed  above,  major  technological,  scientific,  and  con¬ 
ceptual  strides  have  been  made  towards  developing  real- 
world  neuroimaging  as  a  viable  and  legitimate  approach  to 
research  and  technology  application  development.  Still,  the 
creation  of  integrated,  fieldable,  wearable,  multi-aspect  mea¬ 
surement  systems  and  the  attendant  capabilities  to  enable  the 
efficient  handling,  analysis,  and  modeling  of  real-world  neu¬ 
roimaging  data,  and  the  use  of  such  data  in  systems  designs, 
will  require  further  advancements  across  a  wide  range  of 
technology  domains  to  enable  a  fully  realized  capability.  This 
section  delineates  major  challenges  and  targeted  enabling 
technology  capabilities  for  each  of  the  four  critical  technical 
areas  identified  previously:  Neuroimaging  Hardware,  Algo¬ 
rithms,  Interfaces,  and  Experimentation,  and  discusses  recent 
research  and  development  efforts  within  our  programs  that 
have  started  to  address  these  challenges. 

A.  NEUROIMACINC  HARDWARE 

Recent  technological  advances  have  greatly  increased  our 
capabilities  for  data  acquisition  and  measurement  of  all 
manner  of  variables  outside  of  the  laboratory.  Eor  neu¬ 
roimaging,  a  number  of  commercially-available  devices  are 
starting  to  make  fieldable  recording  of  EEG  data  more  reality 
than  science  fiction  (e.g.  ABM  X-series  (Carlsbad,  CA, 
USA),  Emotiv  Epoc,  Quasar  DSI  (San  Diego,  CA,  USA)). 
However,  there  remain  several  interrelated  technical  hurdles 
that  still  must  be  overcome  for  real-world  neuroimaging 
systems  to  truly  provide  either  laboratory-grade  real-world 
measurement  capabilities  or  commercially-viable  end-user 
applications. 

In  the  neurotechnologies  field,  one  of  the  most  active, 
current  research  and  development  areas  to  meet  either  of 
these  objectives  is  in  dry  sensor  designs  that  can  provide 
comparable  signal  quality  as  standard  passive  or  active  sensor 
designs  that  require  conductive  gels  and  extended  application 
time  [54],  [55].  Generally,  conductive  gels  serve  two  primary 
purposes.  The  first  is  to  increase  the  conductive  nature  of  the 
skin,  by  creating  a  moistened  conductive  bridge  with  the  elec¬ 
trode  tip,  lowering  the  overall  impedance  between  the  inter¬ 
nal  biological  source  and  the  recording  system  components. 


This  problem,  however,  has  generally  been  alleviated  by  the 
development  of  ultra-high  impedance  amplifiers  capable  of 
matching  the  conductance  of  dry  skin  [56],  which  can  be  in 
excess  of  megaohms.  Second,  it  provides  a  physical  bridge  to 
easily  penetrate  hair  and  dead  skin,  ensuring  a  strong,  reliable 
connection. 

One  approach  to  dry  sensor  designs  is  to  use  a  soft,  pliable 
pad  which  conforms  to  the  contours  of  the  skin,  providing  a 
large  surface  area  over  which  to  collect  the  electrical  poten¬ 
tial  and  ensuring  a  reliable  connection.  Partners  at  NCTU 
have  shown  that  using  a  conductive  fabric  wrapped  around  a 
pliable  foam  polymer  (see  Eigure  3  for  an  example)  allows 
a  consistent  contact,  even  under  motion.  This  design  also 
alleviates  concerns  about  long-term  comfort,  due  to  the  soft 
nature  of  the  material,  and  poses  minimal  injury  risk  (see 
below;  [40]).  As  a  validation  of  this  approach,  an  example 
comparison  between  a  traditional  gelled  wet  electrode  and 


FIGURE  4.  Data  quality  for  dry  EEG  systems  are  comparable  to  that  of 
conventional  "wet"  gel-based  approaches  for  both  brain  and  muscle 
activity  (top  panels).  Further,  dry  electrodes  have  much  more  stable 
impedance  over  time  as  compared  to  gel  electrodes,  which  tend  to  dry 
out,  increasing  the  contact  impedance  over  time  (bottom,  adapted  from 
[48]). 

a  dry  electrode  pad  is  shown  in  Eigure  4.  The  two  different 
kinds  of  electrodes  were  placed  in  contact  with  a  subject’s 
forehead,  and  the  contact  impedance  was  measured  and  mon¬ 
itored  over  an  extended  period  of  time.  Aside  from  having 
highly  correlated  signals,  as  shown  both  in  time  and  fre¬ 
quency  plots  (Eigure  4,  top  panels),  the  results  showed  that, 
although  impedance  for  the  dry  electrode  was  substantially 
greater  than  for  the  wet  electrode,  it  was  still  in  a  reasonable 
range  for  EEG  measurements.  However,  in  contrast  to  the  wet 
electrode,  which  tended  to  dry  out  such  that  the  impedance 
became  higher  over  time,  the  dry  electrode  remained  stable 
after  several  hours  of  recording.  Note  that  in  this  comparison, 
we  did  not  perform  any  skin  preparation  for  either  style  of 
electrodes.  Due  to  the  large  surface  contact  area,  the  dry  pads 
are  able  to  achieve  relatively  low  contact  impedance  similar 
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to  that  of  traditional  gel  approaches.  Importantly,  this  elec¬ 
trode  style  requires  minimal  preparation,  is  a  cost-effective 
solution,  and  in  conjunction  with  a  high-impedance  amplifier, 
provides  signal  quality  approximately  equivalent  to  that  of 
the  classic  “wet”  electrode  after  preparation,  and  shows  even 
better  performance  than  the  wet  style  over  moderate  and  long 
time  periods  (Figure  4). 

This  technique  provides  comparable  performance  to 
laboratory-grade  wet  electrodes  on  glabrous  skin,  with 
decreases  in  performance  when  attached  to  hairy,  non- 
glabrous  skin  [57].  To  penetrate  hair,  several  variations  of 
spring-loaded  multi-pin  style  electrodes  have  been  developed 
with  signal  characteristics  similar  to  that  of  the  foam  pad 
electrode  and  equivalent  functionality  as  gelled  methods.  In 
this  case,  an  array  of  tiny  pins  penetrates  the  hair,  distributing 
the  connection  across  several  individual  contact  points  [58]. 
In  the  near-term,  this  approach  may  be  more  feasible  for 
systems  designed  for  scientific  study  than  for  fielded  applica¬ 
tions  due  to  cost,  the  potential  for  injury  risk  in  some  settings, 
and  the  potential  for  participant  discomfort  over  long  time 
periods.  Additionally,  because  dry  approaches  rely  solely 
on  mechanical  connection  with  the  scalp,  there  are  open 
questions  about  whether  this  will  be  a  reliable,  consistent 
medium  over  time,  especially  in  high-movement  scenarios. 
Given  this  and  the  potential  for  discomfort  from  the  mechani¬ 
cal  connection,  NCTU,  with  ART  partners,  are  currently  per¬ 
forming  use-case  evaluations  and  exploring  novel  materials 
(see  below)  and  sensor  designs. 

Form  factor  design  will  continue  to  be  a  critical  develop¬ 
ment  area,  because,  in  order  for  any  system  to  be  truly  field- 
able,  user  comfort  and  safety  must  be  a  primary  consideration. 
For  example,  the  system  must  be  comfortable  for  long-term 
wear;  designs  will  need  to  avoid  the  possibility  of  “hot  spots” 
of  discomfort  that  can  form  as  a  result  of  constant  pressure  on 
the  same  locations.  This  likely  means  avoiding  designs  that 
involve  semi-rigid  or  stiff  spring-based  mechanisms  for  hold¬ 
ing  electrodes  in  place,  such  as  the  Emotiv  EPOC  or  Quasar 
DSI  models.  These  approaches  require  pressure  behind  each 
electrode  and  could  be  particularly  problematic  for  larger 
head  sizes,  especially  if  used  in  combination  with  multi-pin 
style  dry  electrodes  that  could  place  pressure  on  very  small 
contact  points  [59].  Similarly,  systems  must  be  as  lightweight 
as  possible  to  minimize  muscle  fatigue,  while  remaining  as 
unobtrusive  to  the  user  as  possible  so  as  not  to  inhibit  their 
inclination  to  wear  the  unit  on  a  regular  basis.  This  includes 
social  issues,  as  well  as  usability,  in  order  to  ensure  consistent 
compliance  across  a  potentially  diverse  user  base. 

Considering  that  these  systems  are  worn  on  the  head,  safety 
becomes  critical  to  putting  neurotechnology  applications  into 
task  environments  where  risks  from  impact  (e.g.  an  automo¬ 
bile  or  bicycle  crash)  are  present.  Protective  helmets  utilize 
deformable  materials,  which  are  designed  to  absorb  energy 
by  deformation,  thus  protecting  the  head  from  trauma.  This 
deformation  requires  both  time  and  space,  and  poses  at  least 
two  problems  for  any  design  involving  hard  components, 
such  as  communications  gear  or  EEC  sensor  mounted  directly 


above  the  scalp,  which  compromises  the  space  between  it  and 
the  helmet  shell.  First,  these  components  become  potential 
sources  of  direct  ballistic  injury  to  the  user  by  transferring  the 
energy  directly  into  the  scalp.  Secondly,  rigid  structures  will 
inhibit  the  intended  deformation  of  the  material,  thus  decreas¬ 
ing  the  helmet’s  effectiveness  in  absorbing  energy  and  allow¬ 
ing  more  to  be  transferred  on  to  the  wearer.  For  electronic 
components  to  be  seamlessly  integrated  with  embedded  pro¬ 
tection  devices  there  must  be  significant  advances  in  the  area 
of  deformable  electronics  in  order  to  minimize  risk  to  the  user. 

To  this  end,  ART  has  been  investigating  the  fabrica¬ 
tion  of  pliable,  stretchable  conductive  polymer  substrates 
as  a  basis  for  electronics  platforms  that  may  serve  as 
the  foundation  for  biological  sensing  [60].  While  recent 
advances  in  material  sciences  have  allowed  the  develop¬ 
ment  of  deformable  conductive  polymers,  one  primary  chal¬ 
lenge  with  the  current  commercially  available  solutions  is 
that  the  conductance  changes  dramatically  as  the  material 
deforms.  This  poses  a  severe  challenge  for  EEG,  which 
targets  extremely  small  fluctuations  in  voltage  against  a 
high-noise  background.  However,  collaborators  in  ARL’s 
Weapons  &  Materials  Research  Directorate  have  developed  a 
unique  process  that  yields  soft  meso-scale  polymer  compos¬ 
ites  capable  of  maintaining  low  levels  of  signal  attenuation 
(<3  dB)  while  subjected  to  biaxial  hyperelastic  stretching 
(See  Figure  5).  Particularly,  by  adjusting  the  manufacturing 
process  to  adjust  the  orientation  behavior  of  carbon-nanotube 
particulates  within  elastomer  substrates,  structures  can  be 
formed  that  maintain  conductivity  even  while  being  stretched 
by  large  amounts.  Meanwhile,  sophisticated  and  unique  new 
electromechanical  characterization  techniques  were  recently 
developed  by  collaborators  in  ARL’s  Vehicle  Technology 
Directorate  that  allow  us,  for  the  first  time,  to  make  accurate 
measurements  of  the  influence  that  very  large  biaxial  strains, 
frequency,  temperature,  and  material  type  have  on  stretchable 
electronic  signal  attenuation  and  filtering  characteristics  [61]. 

Ongoing  efforts  at  ARL,  funded  by  the  ARL  Director’s 
Research  Initiative  [62],  combine  the  expertise  across  three  of 
ARL’s  directorates  in  materials  science  and  neurotechnology 
to  exploit  this  technology  for  EEG  by  exploring  the  use  of 
such  conductive,  deformable  materials  as  a  replacement  for 
the  fixed-conductor  metallic  wire  leads  connecting  electrode 
tips  with  the  primary  electronics  of  the  systems.  Eor  example, 
an  early  prototype  sample  (shown  in  Eigure  5)  replicates 
a  typical  9-electrode  10/20  style  layout,  but  is  capable  of 
stretching  to  match  different  head  sizes  while  causing  min¬ 
imal  signal  attenuation.  Eurther  iterations  focus  on  refining 
the  fabrication  techniques  to  tune  the  material  conductance 
profile,  as  well  as  identifying  the  appropriate  methods  for 
characterizing  and  validating  the  efficacy  of  this  technology 
when  used  for  the  EEG  domain.  While  one  goal  is  integra¬ 
tion  with  the  reconfigurable  suspension  pads  of  a  helmet, 
thus  alleviating  the  concerns  mentioned  above  during  haz¬ 
ardous  situations,  stretchable  or  compressive  polymers  could 
additionally  be  an  ideal  replacement  for  rigid  conductors 
when  using  a  multi-pin  or  bristle  style  electrode,  alleviating 
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the  concerns  about  long-term  comfort  and  safety  discussed 
above. 


FIGURE  5.  Stretchable,  pliable  conductors  for  biosensing.  Example 
prototype  9-sensor  strip  of  stretchable,  pliable  materials  usable  for  EEC 
data  collection  can  stretch  beyond  33%.  This  technology  allows  it  to 
easily  conform  for  differing  head  sizes  (left),  while  also  providing  comfort 
and  safety  of  the  user  when  built  into  head  protection  devices. 

At  the  systems  level,  some  of  the  concerns  found  at 
the  sensor  level  are  also  relevant  (e.g.,  weight,  comfort, 
ruggedization),  while  several  additional  technology  advances 
will  be  needed,  such  as  reducing  power  consumption  and 
increasing  onboard  processing  power  and  data  transmission 
bandwidth.  All  of  these  pervasive  challenges  are  amplified  by 
the  implementation  of  multi-aspect  measurement  approaches 
that  capture  not  only  EEG  data  to  measure  brain  activity,  but 
also  other  sensors  to  capture  additional  data  that  characterize 
non-brain  related  physiological,  behavioral,  and  contextual 
variables. 

The  necessity  for  the  use  of  multiple  measures  stems  from 
concerns  that  are  both  conceptual  and  pragmatic.  Conceptu¬ 
ally,  the  highly  multi-dimensional  nature  of  human  behavior, 
in  general,  and  human  brain  behavior,  specifically,  strongly 
suggests  that  no  single  available  measure  alone  can  provide 
the  information  needed  to  fully  characterize  human  cognitive 
performance  (cf.  [20]).  Moreover,  given  the  highly  integrated 
nature  of  many  of  the  objective  signals  typically  employed, 
such  as  measures  of  central  or  peripheral  physiological  func¬ 
tion,  specific,  behaviorally-relevant  states  are  not  uniquely 
mapped  to  specific  values  of  these  measures.  This  indicates 
that  information  from  other  simultaneously  varying  measures 
of  state  or  context  is  needed  to  disambiguate  the  meaning  of 
the  recorded  signals.  Practically,  in  either  real-world  exper¬ 
imental  paradigms  or  neurotechnology  application  environ¬ 
ments,  performance  of  a  given  task  at  any  given  time  is  not 
typically  known,  making  it  difficult,  if  not  impossible,  to 
assign  often  task-dependent  meanings  to  patterns  of  observed 
measurements.  External  variables  are  needed  for  the  segmen¬ 
tation  of  continuous  data  streams  into  task-relevant  epochs, 
which  are  typically  well-defined  in  the  laboratory.  Choosing 
the  right  combination  of  sensor  technologies  to  adequately 
characterize  the  relevant  aspects  of  real-world  behavior  will 
be  a  critical  challenge,  which  is  complicated  for  real-world 
neuroimaging  systems  by  the  need  to  integrate  and  ade¬ 
quately  synchronize  these  data  streams  under  the  constraints 
of  weight,  power,  data  transmission,  and  cost. 


B.  ALGORITHMS 

Once  neuroimaging  data,  real-world  or  otherwise,  are 
acquired,  extensive  and  often  computationally-intensive  pro¬ 
cessing  is  necessary  before  the  data  are  in  a  form  that  is 
easily  usable  and  suitable  for  the  analysis,  modeling,  and 
interpretation  of  brain  behavior  or  for  the  transformation 
into  metrics  of  user  state  or  performance  in  brain-based 
human-systems  interactions.  Classically,  the  processing  and 
analysis  of  EEG  data  in  research  settings  requires  extensive 
effort  by  trained  experts  who  must  first  visually  inspect  data 
and  identify  non-brain-related  signal  artifacts  or  other,  often 
irrelevant,  sources  of  signal  variance.  This  process  is  time 
consuming  even  in  highly-controlled  laboratory  experiments 
where  task  conditions  are  held  constant  for  task  durations  that 
are  relatively  short,  the  size  of  datasets  is  relatively  small,  and 
task  performance  is  highly  constrained  and  well  annotated  in 
the  data  streams  (e.g.,  time-stamped  event  markers  indicating 
task,  stimulus,  or  response  onsets  and  offsets).  However,  in  a 
approach  to  real-world  neuroimaging  research,  many  of  these 
controls  that  serve  to  constrain  the  data  processing  and  anal¬ 
ysis  problem  in  laboratory-based  research  are  inapplicable. 
Erom  a  scientific  perspective,  one  possible  solution  to  this 
issue  entails  recording  over  much  longer  time  periods  than 
the  typical  laboratory  study,  with  concomitant  increases  in  the 
size  of  experimental  datasets  and  the  costs  of  data  processing. 

Erom  an  application  development  perspective,  the  empha¬ 
sis  for  real-time  or  near  real-time  system  interactions  that 
are  central  to  many  neurotechnology  concepts  obviates  the 
possibility  to  depend  upon  the  extensive,  expert-driven, 
post-hoc  data  processing  typical  of  the  research  laboratory. 
However,  a  common  need  in  real-world  neuroimaging 
research  and  neurotechnology  development  is  the  ability  to 
detect  and  identify  salient  events  whose  occurrence  during 
realistic  behavior  is  not  known  a  priori  in  either  case. 

Consider  the  ubiquitous  example  of  events  that  have  long 
posed  significant  challenges  in  EEG  analysis:  the  presence 
of  motion-  and  muscle-related  signal  artifacts  (for  a  review, 
see  [63]).  In  laboratory  experimentation,  the  presence  of  arti¬ 
fact  components  is  often  reduced  by  asking  participants  to 
minimize  their  body  movements,  though  the  persistent  neces¬ 
sity  for  time-consuming  visual  inspection  and  data  rejection 
would  suggest  that  the  effectiveness  of  such  instructions  is 
far  from  perfect.  Moreover,  during  real-world  neuroimaging 
experiments,  such  constraints  would  be  counter-productive 
to  the  observation  and  study  of  naturalistic  behavior. 
Eor  user  applications,  it  would  be  impractical  to  constrain 
user  movements  in  order  to  achieve  reliable  functionality, 
leading  to  a  potential  lack  of  user  acceptance.  In  either  case, 
an  increase  in  the  likelihood  and  frequency  of  motion-  and 
muscle-related  artifacts  in  these  data  are  expected.  Our  team 
has  achieved  several  advancements  in  algorithm  development 
to  enhance  real-world  neuroimaging  for  both  scientific  and 
applied  pursuits. 

UTS  A  and  ARE  partners  have  used  autoregressive  (AR) 
modeling  techniques  to  create  type-specific  artifact  models. 
This  enables  both  automatic  artifact  detection  and  classifica- 
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tion  [64]  and  offers  a  possible  avenue  for  cleaning  the  data 
ahead  of  subsequent  analysis  or  system  integration. 

To  derive  type-specific  artifact  models  it  is  necessary  to 
obtain  labeled  training  instances  for  each  artifact  type.  The 
results  of  this  research  effort  indicate  that  this  can  be  done  by 
having  the  research  subject  undergo  a  quick  battery  (<  10  min) 
of  calibration  tasks  designed  to  elicit  standard  artifacts  (e.g. 
eye  blinking,  jaw  clenching,  and  head  turning)  prior  to  sys¬ 
tem  use.  Once  the  type-specific  AR  models  have  been  cre¬ 
ated,  the  parameters  of  the  model  are  then  used  as  features 
to  train  a  multi-class  support  vector  machine  (SVM)  [64]. 
This  approach  has  yielded  remarkable  average  classification 
results  >95%  across  the  following  artifact  categories:  eye 
blink,  jaw  clench,  jaw  movement,  head  rotation,  eyebrow 
movement,  left  eye  movements,  upward  eye  movements,  and 
none  (Figure  6).  This  work  represents  an  important  step  in 
objective  artifact  detection  and  extends  previous  research  in 
artifact  classification,  thus  enabling  artifact  to  be  considered 
as  markers  of  salient  events  and  a  critical  component  of 
developing  the  context  of  the  neural  data.  Further,  this  is  a 
relatively  fast  computational  approach  that  is  suitable  to  real¬ 
time  processing  and  applications. 

The  AR  approach  above  has  shown  promise  when  exem¬ 
plar  artifacts  are  known.  However,  it  is  unrealistic  to  assume 
that  all  artifacts  can  be  captured  using  that  approach. 
An  alternative  method  is  to  utilize  measures  that  account  for 
differences  in  the  dynamic  nature  of  brain  versus  non-brain 
signal  sources  and  are  therefore  less  affected  by  the  presence 
of  artifacts  in  the  EEG  signal. 

Recently,  ARE  has  collaborated  with  UM  and  DCS  on 
work  assessing  phase  synchronization  that  has  shown  it  is 
possible  to  detect  statistically  significant  changes  in  neu¬ 
ral  dynamics  during  activities  that  would  have  traditionally 
been  “off-limits”  due  to  artifact  severity  [65],  [44],  [66]. 
Eor  example,  Lau,  et  al.  [44]  demonstrated  that  a  P300-like 
cognitive  response  to  a  visual  oddball  discrimination  task 
could  be  obtained  during  walking  using  the  weighted  phase 
lag  index  (WPLI),  which  is  a  measure  describing  the  phase 
synchronization  between  disparate  neural  sources  recorded 
at  the  scalp  [67].  Phase  synchronization  is  used  to  refer  to 
the  tendency  of  two  (or  more)  oscillators  to  exhibit  consistent 
phase  locking  or  entrainment,  in  response  to  specific  condi¬ 
tions,  such  as  the  establishment  of  a  causal  pathway  or  the 
presence  of  a  common,  third-party  influence  [68].  By  limiting 
analysis  to  only  the  imaginary  portion  of  the  cross-spectrum 
between  two  electrodes,  phase  analysis  methods  such  as 
the  WPLI  are  able  to  strongly  discount  volume  conduction 
effects,  which  result  in  the  mixture  of  the  signals  produced 
by  different  neural  and  non-neural  sources  before  they  reach 
the  sensors  located  on  the  scalp,  giving  the  researcher  a 
clearer  picture  of  true  neural  dynamics.  Gordon,  et  al.  [66] 
extended  this  approach  by  using  the  AR  methods  described 
earlier  to  first  model  the  data  prior  to  computing  phase  syn¬ 
chronization  statistics.  Using  this  technique,  the  authors  were 
able  to  demonstrate  further  resistance  to  EMG  artifacts  by 
analyzing  a  combination  of  simulated  and  real  neural  data. 


Artifact  Classification 
Accuracy 


FIGURE  6.  Performance  of  AR-based  artifact  identification  methods. 
Discrimination  accuracy  is  extremely  high  for  subject-specific  models  of  7 
different  type  of  movement  artifacts  (A).  Independent  features  are  easily 
visualized  when  separated  using  Principle  Component  Analysis  (first  3 
components  shown,  B). 


Importantly,  the  approach  described  here  does  not  rely  on  a 
priori  knowledge  of  what  the  artifacts  may  be  and  are  also 
applicable  to  the  low-density  electrode  montages  expected  in 
many  BCIT  applications. 

These  are  approaches  that  operate  in  “scalp  space.”  Over 
the  past  several  years,  methods  for  blind  source  separation 
(BSS)  have  become  relatively  standard  for  the  separation 
of  brain  and  non-brain  related  signal  components  in  EEG 
data.  Blind  source  separation  methods,  specifically,  ICA 
approaches  developed  by  UCSD  collaborators,  work  by  pro¬ 
jecting  the  scalp  recordings,  which  are  assumed  to  be  a  linear 
mixture  of  a  set  of  latent  source  variables,  into  a  new  source 
space  by  maximizing  statistical  independence  between  the 
source  variables  [69].  This  approach  has  been  shown  to  be 
useful  for  both  scientific  and  applied  pursuits;  however,  there 
are  a  number  of  practical  issues  when  performing  ICA  on 
EEG  data.  Eor  starters,  ICA  requires  that  the  number  of 
sources  be  equal  to  the  number  of  electrodes.  This  assump¬ 
tion  is  unlikely  to  be  true  in  practice  and  thus  the  source 
separation  problem  is  typically  not  well-posed  and  is  often 
under-determined.  The  sources  must  also  be  stationary  and  at 
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most  one  source  can  be  Gaussian.  The  assumption  of  source 
stationarity  is  equally  unlikely,  at  least  over  extended  periods 
of  time  and/or  when  multiple  complex  tasks  are  being  per¬ 
formed.  Finally,  to  capture  the  source  statistics,  the  researcher 
must  know  a  priori  neural  event  boundaries  and  must  possess 
enough  data  to  adequately  represent  the  underlying  processes 
of  interest. 

Despite  these  issues  ICA  has  proven  to  be  an  effective  tool 
for  data  cleaning  and  pre-processing  [70]  and  has  received 
attention  with  respect  to  isolating  various  neural  processes  of 
interest  [71].  UCSD  collaborators  are  working  on  a  number  of 
extensions  proposed  to  improve  the  performance  of  standard 
ICA  techniques,  similar  to  what  has  been  advocated  by  other 
groups  [72].  These  include  adding  the  capacity  for  multiple 
mixture  [73]  and  adaptive  mixture  models  [74]  that  use  meth¬ 
ods,  such  as  maximum  likelihood  estimation,  to  produce  the 
most  likely  decomposition  at  different  time  points.  In  addi¬ 
tion,  clustering  methods  and  machine  learning  approaches 
have  been  used  to  identify  and  group  relatively  stable  com¬ 
ponents  across  task  conditions  and  recording  sessions  [75], 
[76].  In  this  way,  rather  than  assuming  all  components  from 
a  specific  decomposition  are  unique  and  equally  informative, 
researchers  are  able  to  better  identify  relationships  between 
changes  in  context  and  changes  occurring  in  seemingly  inde¬ 
pendent  neural  processes. 

To  further  combat  some  of  the  many  issues  related  to 
processing  EEG,  our  multi-aspect  approach  looks  beyond  just 
EEG.  By  simultaneously  recording  multi-aspect  physiologi¬ 
cal  data,  researchers  can  better  understand  the  subtleties  of  the 
brain-body-behavior  relationship  [16].  This  also  facilitates 
the  development  of  more  robust  tools  by  providing  secondary 
sources  of  information  that  can  be  leveraged  when  and  as 
needed  to  develop  the  context  for  interpreting  measured  neu¬ 
ral  activity.  Currently,  collaboration  between  DCS  and  ARE 
is  underway  to  simultaneously  record  high-density  EEG, 
EMG  of  the  face,  neck  and  shoulders,  eye  tracking,  facial 
expression  tracking,  and  upper  limb  motion  tracking.  Similar 
efforts  are  underway  at  partner  locations  in  UCSD,  UM,  and 
NCTU.  The  goal  of  these  efforts  is  to  derive  relationships  that 
can  be  used  to  fuel  the  systems-level  development  of  neu¬ 
rotechnologies  that  then  can  be  reliably  situated  in  real-world 
contexts. 

We  are  also  looking  towards  the  development  of  more 
real-time  processing  techniques.  Low-pass  and  high-pass  fil¬ 
ters  can  be  implemented  in  hardware  or  through  specific 
software  applications.  AR  methods  such  as  those  previously 
described  can  be  applied  quickly  and  efficiently  and  many 
phase  synchronization  approaches  have  online  extensions  that 
do  not  require  epoching  of  the  data,  but  rather  use  the  Hilbert 
transform  to  derive  a  more  continuous  notion  of  phase  [77]. 
Our  partners  have  also  worked  on  hardware-based  ICA  appli¬ 
cations  [78]. 

Significant  progress  has  been  made  by  BCI  developers, 
who  have  viewed  standard  EEG  processing  as  forms  of 
feature  extraction  that  then  could  be  used  to  feed  state- 
classification  tools.  These  tools  range  from  relatively  stock 


approaches,  such  as  Linear  Discriminant  Analysis  (EDA), 
Common  Spatial  Patterns  (CSP),  or  Support- Vector  Machines 
(SVM),  that  operate  on  minimally  processed  data,  to  more 
in-depth  designs,  such  as  those  from  our  collaborators  [79], 
[80],  in  which  complex,  multi-tiered  pre-processing  tech¬ 
niques  and/or  event-related  analyses  are  leveraged  in  conjunc¬ 
tion  with  standard  machine  learning  approaches.  These  latter 
approaches  represent  a  significant  step  towards  the  types  of 
capabilities  needed  for  current  BCIT  development.  Rather 
than  taking  an  agnostic  view  of  the  information  contained  in 
the  EEG  signals  and  attempting  to  perform  state  classification 
on  the  raw  or  minimally  processed  data,  these  neurotech¬ 
nology  developers  are  taking  a  systems-level  view  in  which 
multiple  components  each  enable  the  extraction  of  different 
information  from  the  imaged  neural  data.  To  support  this 
development,  engineers  and  scientists  working  at  UCSD  have 
been  developing  a  tool,  known  as  BCILAB,  that  enables  rapid 
deployment  of  a  number  of  EEG  processing  pipelines  coupled 
with  state-of-the-art  classification  systems  [81]. 

C  INTERFACES 

One  significant,  if  perhaps  insufficiently  appreciated,  chal¬ 
lenge  to  the  real-world  neuroimaging  approach  will  be  the 
research  into  and  the  design  and  development  of  interfaces 
that  can  enable  minimally-intrusive  sampling  of  participant 
behaviors  while  they  are  observed  in  natural  conditions,  and 
the  effective  interaction  of  researchers  and  developers  with 
the  large  and  complex  datasets  that  will  result  from  these 
research  efforts.  Thus,  the  requirements  for  user  interfaces 
will  be  significantly  altered  for  a  real  world  neuroimaging 
system  compared  to  conventional  neuroimaging  systems.  One 
of  the  main  changes  will  be  the  shift  in  responsibility  for 
monitoring  the  hardware  and  data  collection  from  an  expert 
who  is  not  wearing  the  system  to  a  wearer  with  limited  or 
potentially  unknown  level  of  expertise. 

In  SEN  System  V 1 .0,  ARE  and  DCS  have  designed  a  range 
of  applications  for  the  Android  platform  on  cell  phones  and 
tablet  computers.  The  main  feature  of  the  system  is  a  set  of 
widgets  on  the  home  screen  that  allow  the  user  to  self-report 
the  start  and  end  of  pre-defined  activities  (eating,  drinking, 
meeting,  conversations,  etc.),  as  well  as  unexpected  events 
(startling  sounds,  forgot  to  log  the  start  or  end  of  an  event, 
adjusted  the  fit  or  positioning  of  measurement  equipment, 
etc.).  There  is  also  a  master  scheduler  application,  which  is 
nearly  invisible  to  the  user.  This  application  can  trigger  the 
start  of  self-report  surveys  and  experimental  tasks  based  on 
a  preset  schedule.  We  have  converted  a  number  of  published 
self-report  surveys,  such  as  the  Karolinska  Sleepiness  Survey 
[82],  the  task  induced  fatigue  scale  [83],  the  NASA  TLX 
[84],  [85],  and  visual  analog  scales  of  stress  and  fatigue  [86], 
[87] ,  into  touch-screen  versions  that  can  be  completed  quickly 
with  minimal  impact  on  the  user’s  work  or  everyday  tasks 
and  behavior.  In  the  near  future,  we  plan  to  enable  automatic 
processing  of  the  incoming  data  in  order  to  trigger  and  aid 
in  troubleshooting  the  physiological  systems  when  they  start 
to  fail.  These  applications,  which  are  undergoing  use-case 
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evaluations,  all  assist  in  transitioning  data  collection  and 
monitoring  capabilities  from  an  experimenter  to  the  user.  Data 
collected  with  SFN  System  Vl.O  will  aid  our  development  of 
future  versions  of  the  system  in  which  we  aim  to  implement 
automatic  processing  of  the  physiological  data  streams  to 
further  remove  the  need  to  intrusively  sample  subjective  data. 

A  second  critical  challenge  is  enabling  an  effective 
interaction  between  scientists,  engineers,  and  technology 
innovators  to  analyze,  interpret,  and  ultimately  create  novel 
technologies  from  the  complex  multi-aspect  datasets  created 
with  these  technologies.  UCSD  continues  to  evolve  their 
EEGLAB  graphical-user  interface  to  incorporate  many  of 
the  aforementioned  advances  in  algorithms  and  to  enable 
EEG  specialists  to  more  readily  access  advanced  signal  pro¬ 
cessing  techniques.  Similarly,  UTSA  and  ARE  have  part¬ 
nered  to  implement  the  DETECT  toolbox  for  MATLAB 
that  offers  the  ability  for  EEG  specialists  to  access  the  AR 
models  described  above  [37],  [38].  Einally,  as  described 
above,  UCSD’s  BCILAB  was  specifically  developed  for  non- 
EEG  specialists  (Christian  Kothe,  personal  communications). 
While  important  progress  has  been  made,  the  area  of  interface 
development  for  the  broader  community  remains  a  significant 
challenge. 

D.  EXPERIMENTATION,  TESTING,  AND  VALIDATION 

Above  we  highlighted  three  different  initial  approaches 
that  have  been  taken  to  extending  neuroimaging  into 
real-world  settings:  examining  human  behavior  in  relatively 
well-controlled  real-world  task  environments;  examining 
real-world  tasks  under  controlled  laboratory  conditions;  and 
examining  laboratory  tasks  in  more  realistic  task  conditions. 
All  of  these  approaches  enable  significant  advances,  both 
scientifically  and  technically,  towards  the  real-world  neu¬ 
roimaging  vision  laid  out  here.  However,  the  maturation  of 
enabling  capabilities  discussed  here,  as  well  as  the  growing 
ubiquity  of  sensor-based  user  technologies  in  our  everyday 
lives,  is  opening  up  new  possibilities  for  the  study  of  human 
neural  behavior  in  its  natural  state.  In  particular,  these  tech¬ 
nologies  are  envisioned  to  enable  more  ecological  approaches 
utilized  in  other  disciplines  in  the  human  and  behavioral 
sciences-approaches  that,  by  contrast,  have  not  been  prevalent 
in  the  cognitive  neurosciences-to  both  the  observation  and  the 
experimental  study  of  real-world  brain  behavior. 

Erom  both  the  scientific  and  applied  perspectives,  our  cur¬ 
rent  efforts  focus  on  understanding  and  developing  technolo¬ 
gies  that  are  tailored  to  individual  users.  This  approach  is 
feasible  in  large  part  due  to  the  recent  re-emergence  of  a 
focus  on  individual  differences  in  the  field  of  cognitive  neu¬ 
roscience  (e.g.,  see  [88],  [89])  coupled  with  the  remarkable 
advances  observed  in  the  application  of  machine  learning  and 
classifications  approaches  to  individualizing  BCIT  applica¬ 
tions  over  the  past  decade.  These  advances  are  making  subtle 
but  critical  shifts  in  the  paradigms  researchers  are  employing 
to  understand  brain  function.  To  this  end,  we  are  employing 
paradigms  of  large  data  collection  efforts  that  capture  both 
sufficient  real-world  behaviors  and  large  participant  popu¬ 


lations.  Like  many  within  the  cognitive  neuroscience  com¬ 
munity,  our  focus  is  not  only  on  understanding  similarities 
between  people,  but  also  on  uncovering  the  differences  that 
make  people  unique.  We  aim  to  further  this  paradigm  and 
focus  on  understanding  the  differences  in  neural  processing 
between  events  within  individuals.  Our  primary  focus  is  on 
developing  capabilities  for  neuroimaging  in  actual  real-world 
settings;  however,  one  of  the  bridging  technologies  that  may 
further  our  team’s  scientific  objectives  and  enable  the  devel¬ 
opment  of  future  systems  is  virtual  reality  (VR).  Our  general 
approach  outlined  here,  when  merged  with  VR,  will  intro¬ 
duce  new  and  potentially  insightful  cognitive  neuroscience 
experimental  paradigms.  Still,  further  tool  development  in 
VR  is  needed  to  instantiate  this  vision.  To  meet  these  needs, 
ARE  has  been  developing  a  novel  software  framework  for  the 
intelligent  control  of  dynamic,  complex,  high-resolution  VR 
environments.  The  Graph-based  Interactive  Narrative  (GIN) 
system  [46]  utilizes  a  three-layered  hierarchical  graph  frame¬ 
work  that  models  the  physical  topography  of  the  environ¬ 
ment  as  a  set  of  nodes  (i.e.,  path  intersections)  and  edges 
(i.e.,  paths  the  user  can  traverse  between  nodes),  and  allows 
the  monitoring  of  the  participant’s  movement  through  the 
graph  and  the  manipulating  of  stimulus  presentations  relative 
to  the  participant’s  chosen  path.  The  GIN  system  aims  to: 
1)  provide  experimental  participants  with  an  increased  ability 
to  freely  control  their  navigation  throughout  the  virtual  world 
(i.e.,  increasing  perceptions  of  agency  [90]);  2)  maintaining 
experimental  control  by  adapting  the  presentation  of  stimulus 
events  to  the  participants  chosen  path  while  adhering  to  the 
requirements  of  the  experimental  design;  and  3)  improv¬ 
ing  the  tractability  of  the  development  of  VR  scenarios  for 
experimentation  by  providing  capabilities  for  automatically 
controlling  the  placement  and  occurrence  of  stimulus  events. 
The  primary  rationale  for  addressing  this  last  point  stems 
from  the  time-  and  labor-intensity  inherent  to  designing, 
developing,  and  implementing  experimental  VR  paradigms 
that  capture  the  relevant  contextual  effects  on  human  tasks, 
actions,  and  environmental  actions  (cf.  [45]). 

As  a  complimentary  approach,  ARE  partners  at  UCSD 
have  developed  a  new  software  framework,  the 
Simulation  and  Neuroscience  Application  Platform  (SNAP) 
(https://github.com/sccn/SNAP),  that  aims  to  provide  new 
rapid  prototyping,  testing,  and  real-time  control  of  com¬ 
plex  tasks  and  paradigms  for  implementation  in  real-world 
neuroimaging  experimental  designs.  The  SNAP  framework 
is  integrated  with  a  commercial-grade  C-h-h  game  engine, 
Panda3D,  to  enable  the  efficient  transition  from  controlling 
simple  2-D  psychological  stimuli  to  running  complex  experi¬ 
mental  tasks  involving  full,  immersive  3-D  visual  displays. 
Similar  capabilities,  for  example  implemented  on  portable 
interface  platforms  (see  “Interfaces,”  above),  could  support 
the  implementation  of  experimental  paradigms  in  real-world 
environments. 

In  addition  to  novel  paradigms,  as  fieldable  neuroimaging 
expands  and  new  hardware  approaches  are  developed,  it  will 
be  necessary  to  continually  evaluate  and  compare  the  efficacy 
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of  new  technologies.  For  example,  recent  ARL  tests  of  the 
timing  accuracy  and  variance  in  trigger  integration  associated 
with  several  different  commercial  EEG  systems  has  revealed 
strong  timing  errors  in  certain  products  188],  191].  Similarly, 
comparisons  of  system  design  features  that  directly  affect 
overall  usability,  such  as  average  application  time,  adaptabil¬ 
ity  for  different  head  sizes,  user  comfort,  or  electrode  stability 
over  time,  have  led  to  a  stronger  understanding  of  which  ele¬ 
ments  are  most  critical  for  application  to  systems  targeted  for 
real-world  use  and  are  aiding  future  technology  development 
159].  One  of  the  critical  issues  with  real-world  neurotechnol¬ 
ogy  testing  and  evaluation  (T&E)  is  the  lack  of  current  formal 
standards  or  procedures  for  EEG  sensors  and  systems  (i.e., 
no  T&E  standards  have  been  adopted  by  IEEE  or  ASTM 
International).  The  most  common  approaches  to  T&E  rely 
on  humans  to  provide  the  native  signals  (e.g.,  see  Eigure  4), 
which  is  sub-optimal  due  to  the  inherent  variability  and  incon¬ 
sistency  of  human  EEG  signals.  Eurther,  while  this  approach 
may  be  acceptable  for  evaluating  certain  aspects  of  sensor 
design,  it  is  more  problematic  for  evaluating  entire  systems. 

Our  team  has  addressed  this  particular  problem  by  attempt¬ 
ing  to  develop  and  validate  an  EEG  “phantom”  device,  which 
could  be  used  to  replace  the  human  head  as  an  objective 
fixture  to  generate  reference  signals  for  these  kinds  of  tests 
(Eigure  7).  Specifically,  the  phantom  serves  as  a  standard, 
known-quantity  signal  that  can  be  used  with  a  putative  sys¬ 
tem,  allowing  an  experimenter  to  quantify  any  differences 
in  the  outputs  of  the  recording  system.  The  physical  prop¬ 
erties  of  the  phantom  are  designed  to  emulate  the  conduc¬ 
tive  properties  of  brain  matter,  skull,  and  skin.  And  while 
there  currently  are  a  number  of  computational  models  that 
simulate  these  properties  of  the  head,  no  physical  models  are 
commercially  available  that  also  have  analogous  conductive 
properties.  Likewise,  the  few  physical,  conductive  models 
or  “phantoms”  discussed  in  the  academic  literature  (e.g. 
[95]-198])  have  not  been  created  in  an  appropriate  form  fac¬ 
tor  for  use  with  complete  conventional  EEG  systems.  Being 
structured  as  such  would  enable  a  host  of  applications,  includ¬ 
ing:  use  as  a  benchmark  for  typical  off-the-shelf  cap-based 
EEG  systems,  assessing  the  impact  of  particular  environments 
on  signal  recording  quality,  and  validating  the  efficacy  of 
putative  signal  detection  or  discrimination  algorithms. 

In  order  to  quickly  advance  the  development  of  this  capa¬ 
bility,  ARL  proposed  and  awarded  an  SBIR  project  topic 
(“Neurological  Simulator  for  Applied  Data  Collection,” 
Army  SBIR  A 10-066)  for  a  phantom  model  that  has  physical 
and  electrical  properties  analogous  to  the  human  head  with 
multiple  sources  for  creating  EEG-like  signals  from  distinct, 
interior  points.  Initial  prototypes  developed  under  this  pro¬ 
gram  have  ranged  from  use  of  simple  conductive  paints  over 
a  plastic  shell  (Physical  Optics  Corp  (Torrence,  CA,  USA); 
CED  Research  Corp.  (Huntsville,  AL,  USA)),  to  varying  mix¬ 
tures  of  carbon  nanotube  polymers  doped  to  match  the  mean 
conductance  of  the  human  head  (Physical  Optics  Corp  199] ; 
Creare,  Inc  (Hanover,  NH,  USA)),  and  even  a  3-layer  realistic 
model  based  on  an  MRI  of  an  actual  head  with  embedded 


dipole  source  points  (Creare,  Inc,  1100]).  Even  though  the 
initial  attempts  fall  short  of  the  necessary  conductive  realism 
to  serve  as  a  universal  standard,  models  developed  thus  far 
have  already  been  found  useful  for  evaluating  system  latency 
189].  Ongoing  efforts  focus  on  improving  fabrication  tech¬ 
niques  to  ensure  reproducibility  of  the  device  while  achieving 
conductive  realism  across  the  entire  “head”  structure. 


FIGURE  7.  Sample  prototype  ""phantom  head"  devices  developed  under 
the  SBIR  program.  Models  provided  by  CFDRC  (left),  POC  (center),  and 
Creare  (right)  are  designed  to  be  conductively  analogous  to  the  human 
head,  with  embedded  sources  to  create  realistic  electrical  dipoles 
detectable  at  the  surface. 


VI.  MOVING  TOWARD  APPLICATIONS 

One  of  the  main  goals  of  developing  real-world  neuroimag¬ 
ing  techniques  is  to  enable  the  development  of  BCITs  that 
can  be  implemented  in  realistic  scenarios.  An  example  of 
such  efforts  is  an  SAIC-University  of  California,  Santa 
Barbara  (UCSB)-ARL  collaboration  that  aims  to  demon¬ 
strate  the  effective  extension  of  a  rapid  serial  visual  pre¬ 
sentation  (RSVP)  based-BCIT  to  an  operationally-relevant 
context  147].  Recent  advances  in  the  application  of  signal 
processing  and  classification  algorithms  and  techniques  to 
EEG  signals  have  enabled  the  development  of  an  RSVP 
BCIT,  which  lets  operators  rapidly  review  or  triage  visual 
imagery  without  requiring  a  manual  response  to  each  image 
1101].  Various  groups  have  shown  that  this  approach  can  be 
used  to  make  the  process  of  identifying  targets  in  large  ensem¬ 
bles  of  images  an  order  of  magnitude  faster  than  a  manual 
search  (e.g.,  see  efforts  by  ARL  partners,  147]).  However, 
initial  applications  typically  used  obvious  target  objects  and 
the  only  task  required  of  the  operator  was  to  focus  on  the 
image  presentation. 

In  order  to  quickly  advance  the  development  of  this  capa¬ 
bility,  ARL  successfully  proposed  a  SBIR  topic  (“Neurologi¬ 
cal  Simulator  for  applied  Data  Collection,”  Army  SBIR  AlO- 
066),  with  three  Phase  I  projects  awarded,  and  one  subsequent 
Phase  II  project  awarded.  This  project  is  aimed  at  developing 
a  phantom  model  that  has  physical  and  electrical  proper- 
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ties  analogous  to  the  human  head,  with  multiple  sources  for 
creating  EEG-like  signals  from  distinct,  interior  points.  The 
RSVP  BCIT  was  incorporated  into  the  simulation  by  having 
an  RSVP  window  pop-up  at  various  times  throughout  the 
mission.  Operators  had  to  switch  from  what  they  were  doing 
and  engage  the  BCIT.  During  the  RSVP  presentation,  each 
image  was  sorted  based  on  an  interest  score  derived  from  the 
evoked  brain  response.  The  operator  was  then  presented  with 
the  images  that  generated  the  highest  interest  score  and  they 
manually  verified  which  of  the  top  scoring  image(s)  contained 
targets. 

The  initial  development  of  this  simulation  environment 
was  focused  on  identifying  the  optimal  parameters  and 
approach  for  classification  of  the  neural  response  in  an 
RSVP  paradigm.  This  included  presentation  properties  such 
as  size,  eccentricity,  and  rate  [104],  as  well  as  the  effects  of 
changes  in  attentional  state  on  classification  accuracy  [105]. 
In  addition,  early  studies  examined  the  effect  of  operator 
multitasking  on  system  performance  [106].  After  the  optimal 
design  parameters  were  identified,  a  simulation  environment 
was  developed  to  quantify  the  benefit  of  incorporating  neural 
processing  techniques  into  a  relevant  operational  context. 
Preliminary  results  indicate  that  the  RSVP  neural  classifica¬ 
tion  scheme  can  dramatically  reduce  the  target  search  time 
while  maintaining  the  accuracy  level  achieved  by  traditional 
manual  search  methods.  Currently,  the  simulation  environ¬ 
ment  (called  RS  VP-based  Adaptive  Virtual  Environment  with 
Neural-processing  or  RAVEN)  is  undergoing  final  implemen¬ 
tation  and  testing  that  focuses  on  the  effect  of  multitasking 
and  task  difficulty  on  overall  system  performance. 


RAVEN  System 


Operator  Screen 

^.ensor  Banner 


Sensor  Portal  Mission  Map 


Model 


Driving 


RSVP 


FIGURE  8.  RSVP-based  Adaptive  Virtual  Environment  with 
Neural-processing  (RAVEN)  system.  RAVEN  system  main  operator  screen 
and  RSVP  window.  The  primary  task  is  identification  of  dismounts  while 
the  vehicle  is  navigating  a  simulated  environment.  Secondary  tasks 
include  identification  of  potential  lED  locations,  monitoring  and 
responding  to  communications  (audio  and  text).  The  BCI  component  is 
engaged  during  RSVP  search  (Mode  3)  and  top  scoring  images  are 
presented  on  the  main  operator  screen. 


ARE,  and  its  partners  at  Teledyne  Scientific  &  Imag¬ 
ing,  SAIC,  DCS,  NCTU,  UCSD,  and  the  U.S.  Army 
Tank  Automotive  Research,  Development,  and  Engineering 
Center  are  attempting  a  similar  effort  to  extend  NCTU-  and 
UCSD-developed  driving  performance  prediction  and  mit¬ 
igation  technologies  into  realistic  driving  scenarios.  Early 


fatigue-based  monitoring  neurotechnologies  demonstrated 
that  high  correlations  could  be  obtained  between  EEC  activity 
and  lane  deviation  performance  on  a  simplified  driving  task 
under  static  and  motion  conditions  [107]-[109].  Our  team 
further  extended  this  effort  to  demonstrate  that  auditory  mit¬ 
igations  could  be  used  to  aid  performance  in  this  scenario 
[1 10],  [1 1 1].  Currently,  we  are  systematically  extending  these 
efforts  to  more  complex  and  realistic  scenarios  including 
adding:  road,  traffic,  and  environmental  complexity;  multi¬ 
tasking;  and  vehicle  convoy  operations. 

Efforts  such  as  the  applications  illustrated  above  aid  in 
understanding  the  practical  aspects  of  real-world  neuroimag¬ 
ing  implementation  and  give  insights  into  many  of  the  con¬ 
ceptual  issues  discussed  herein.  They  also  just  hint  at  the  pos¬ 
sible  brain-based  technologies  that  will  allow  computers,  for 
the  first  time,  to  generate  and  leverage  sophisticated  analyses 
of  the  emotional  and  cognitive  states  and  processes  of  the  peo¬ 
ple  using  them.  These  analyses  have  the  potential  to  radically 
change  our  current  insights  into  basic  human  brain  function 
and  to  revolutionize  the  basic  interactions  people  have,  not 
only  with  the  systems  they  use,  but  also  with  each  other. 
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